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C*~) ■ Abstract. We study the Hopfield model on a random graph in scaling regimes where 

. the average number of connections per neuron is a finite number and where the spin 

\ dynamics is governed by a synchronous execution of the microscopic update rule (Little- 



Oh! 



Hopfield model) . We solve this model within replica symmetry and by using bifurcation 
, analysis we prove that the spin-glass/paramagnetic and the retrieval/paramagnetic 

00 ' transition lines of our phase diagram are identical to those of sequential dynamics. 

■ The first-order retrieval/spin- glass transition line follows by direct evaluation of our 

observables using population dynamics. Within the accuracy of numerical precision 
I ' and for sufficiently small values of the connectivity parameter we find that this 

. line coincides with the corresponding sequential one. Comparison with simulation 

l/"*) ' experiments shows excellent agreement. 

; 

. 

ON ■ 

o : 

1. Introduction 

Since the first analytical work describing pattern recall was presented and the theoretical 
foundations describing the operation of neural networks were subsequently set, the 
progress in the field of attractor neural network models has been advancing revealing 
interesting properties. Early analytical attempts to solve non-trivial neural network 
systems were analytically constrained to the study of fully-connected ones. From a 
| real (i.e. biological) point of view one would ideally prefer to study models of sparse 
connectivity. Subsequently, theoretical work turned (among other directions) to sparse 
models, and, in particular, to the so-called extremely diluted ones PQ. These could be 
still solved exactly owing to the simplification of considering system sizes exponentially 
bigger than the average connectivity per neuron (while both of these numbers were 
eventually sent to infinity). 

One direction in developing the neural network theory further towards realism 
implies moving away from the simplifications made in extremely diluted systems and 
considering finite degrees of connectivity. This however appears to be a highly non- 
trivial step ahead since at an early stage of solving the relevant equations one is 
confronted with more than the traditional two observables (the magnetisation and the 
overlap), in fact, one is required to consider an order-parameter function. 

Although initial attempts to solve such models showed the underlying analytical 
complexity [21 EI, the more refined mathematical tools were only relatively recently 
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developed and in fact for applications different than neural networks (for instance, for 
optimisation problems jlj or error-correcting codes j3])- The first study which applied 
the finite-connectivity methodology to neural network problems has been very recently 
presented in [6^ , where phase diagrams were presented for a variety of synaptic kernels. 
There, the authors study neural networks in which the microscopic neuronal dynamics 
is a sequential execution of an update rule (describing alignment to post-synaptic 
potentials). In this paper we examine the effect of choosing a synchronous microscopic 
dynamics in Hopfield models of finite connectivity, i.e. one in which neurons are updated 
in parallel at each time step. These two different types of dynamics are known to 
share very interesting properties. For instance, in simple ferromagnetic models one can 
prove that thermodynamic observables become identical [7 . It is yet unclear to what 
extent the two types of dynamical models share these common equilibrium features. For 
example, it is known that the phase diagram of the synchronous Hopfield model changes 
[HI E| t whereas that of the Sherrington-Kirkpatrick model remains unaffected [TU] . 

This paper is organised as follows: in the following section we present definitions 
of our model. In section El we derive the saddle point equations for the free energy and 
make our replica-symmetric ansatz. In section 0] we give expressions of the free energy 
in terms of the replica symmetric order function whereas our results and phase diagrams 
are given in section 

2. Definitions 

We study neural network models of N binary neurons er = (01, . . . ,a N ) with o~i = — 1 
representing 'at rest' and o~i = 1 the 'firing' state. The miscroscopic neuron dynamics is 
a stochastic alignment to 'local fields' (the post-synaptic potentials) in which updates 
in neuronal states are made for alii G {1, . . . , N} in a fully synchronous way at each 
discrete time step: 

Probst + 1) = ±1] = -[1 ± tanh(/3^(cr(i)))] (1) 

where hi(<r(t)) = ^ • JijO-j(t). The parameter ft G [0, oo) controls the amount of thermal 
noise in the dynamics with ft = oo corresponding to a fully deterministic execution of 
and ft = to a fully random execution. The quantities describe interaction 
couplings. If one expresses (JIJ in terms of the microscopic state probabilities pt(<x): 

then, for any finite ft and finite N the process (J2J) is ergodic and evolves to a unique 
distribution p^er). It can be shown that this is a unique equilibrium state (obeying 

| This difference seems to be an artifact of the replica symmetric approximation. The two phase 
diagrams become identical within full Replica Symmetry Breaking 8 , although it is unclear at which 
stage of the breaking scheme this occurs. 
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detailed balance) if and only if Jy = Jji [14 §. At equilibrium, the microscopic state 
probabilities acquire the form p^ ~ exp[— (3H(<t)], where 

H W = E lQ g[ 2 awh^Cff))] (3) 

is the Hamiltonian. For the interactions Jy we now take: 

^=^£tftf (4) 

This corresponds to storing p memories £ 4 = , . . . , £?) G {— 1, 1} P among the synapses 
in a Hebbian-type way. The variables G {0, 1} with c^- = Cjj represent dilution, while 
c = Ylj c ij (f° r an corresponds to the average number of connections per neuron. 
Models of the type (j3J) with c —>■ oo (while c ~ log N) are known as extremely diluted 
and due to their simplicity have been studied extensively (see for instance [TTJ[T2J[T3] and 
references therein). What is less known are properties of these systems in the non-trivial 
scaling regime of finite connectivity where c ~ 0(1) (with the probability c/N — > 0). 
Due to the complicated methodology required such systems have only recently been 
studied in |Hj where a thorough bifurcation analysis was performed and phase diagrams 
were derived. For the distribution of the random variable qj we will consider 

PiPii) = JfSev,! +{ 1 -Jj) 5 ^,o (5) 
for all pairs All thermodynamic quantities will have to be averaged over (|SJ). Note 

that due to the system's sparse connectivity the number of patterns can only be finite. 

To derive observable quantities we will calculate the free energy per neuron / = 
— limAr^oo log e~^ H ^ ){dj}, with (• • •) denoting average over the distribution 
of dilution variables. As in all synchronous dynamics models, the evaluation of the 
free energy is greatly simplified by introducing an extra set of spins so that / can be 
rewritten as 

1 ^ e-Wrt) H{a,T) = -Y t a i J i f i (6) 

will be the starting point for our analysis. 




3. Saddle-point equations 



In order to calculate the free energy (JBj) we begin by invoking the replica identity 
(logZ) = lim n ^o _ log(Z n ). One can then take the average over the dilution variables 
resulting in 



f = — lim lim log exp 



c 

2N 



E 



Ki-«j)E (<f^+W) 



(7) 



§ Note that unlike sequential models where detailed balance requires the exclusion of self-interactions, 
this is not here a prerequisite. However, due to the scaling regime of finite connectivity terms originating 
from the self-interacting part will for N — -> oo give a vanishing contribution to thermodynamic 
quantities. 
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where ^ • £^ = EuCfCf- Note that at this stage an exponential has appeared as 
an argument of another exponential. Here, effectively, the inner-most exponential 
introduces an infinite number of observables. However, unlike systems of full- 
connectivity or of extreme dilution where only the linear and quadratic observables 
(order parameters) survive in the limit N — > oo, all cumulants play here a role. They 
can be re-cast in an order-parameter function. This complication is a direct consequence 
of the scaling regime of finite connectivity. 

It is convenient now to use the concept of sublattices jpTO which divide the space of 
sites into 2 P groups 1$ = = £}. We define cr^ = [a], . . . , erf) and abbreviate 

the averages over the sublattices by (P(£))s = J2^P^P(^) "with the probabilities 
p^ = \I^\/N. Then the free energy becomes: 



lim 

N,n (3Nn 



—log y y 



exp 



2N EEE 

era' tt 1 



(8) 



x 226^5^ y cv.^cv,^ 

We now see that an order-function has emerged which we introduce into our expression 
via 



1 = / II \ dp «( CT > r ) 6 p «( ct > r ) - rn E \ 



(9) 



In the above expression we replace the ^-function by its integral representation (which 
introduces the conjugate order function parameter P^(ct,t)). We then perform the 
trace over the spin variables and take the limit iV — > oo in our equations resulting in 
the extremisation problem: 



; = " & ^ Ext v,n \ E p ^ r )^(°"' r ^ + < log E e ~^ (<T,x) ] >« 



+-EE P €( <T ' T )^'(' T, ' T ') et(^')E Q (^ +CT ^)_ 1 ) 



(10) 
(11) 



Variation with respect to the densities P^(cr,r) and P^(<r, r) gives the self-consistent 
equation 



exp 




>* 




E CT ' T ' ex P 


C(E CT » T " ^"(^ T") ( e !(«")EJ<r»+<rA) 







(12) 



and also allows us to eliminate P^(ct,t) from the expression of the free energy which 
now becomes: 

/ = lim^-Extr {^yyPd<r,T)Pe(<r',T>) ( e K^E>^ + ^) _ ^ 

" ^ I CT-r o-'t' 
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Pg(<T',T') (e^'^J««<+<T a ) 



(13) 



This expression requires knowledge of the densities P^(ct,t). However, the evaluation 
of the self-consistent equation (fT2|) is an impossible task unless a further simplification 
is made. To proceed further we will make an assumption about the form of the densities 
P^((T,t). In the spirit of replica symmetry (RS) we will consider that permutation of 
spins within different replicas leave the order parameters invariant. Here however, due 
to the presence of two species of spins we will also require that within the same replica 
group permutation of the states of a a and r a also leave for all a P^(cr, r) invariant: 

PJtr, T ) = [ ,^ rd \, We(h,r,t) (14) 
§v ' J [M(h,r,t)] n 4V 1 v 1 

where N(h, r, t) is the corresponding normalisation constant 

J\f(h, r,t) =4 cosh(/3/i) cosh(/5r) cosh(/3t) + 4 sxnh((3h) sinh(/3r) wah(J3t) (15) 

Let us finally turn to our system's macroscopic observables. Our replicated 
sublattice overlaps will be given by 

mf = <£"m£ a > € (16) 

with the sublattice magnetisations defined as 

A, a 



J2 P d°iT)\ a (17) 



with A = a, t. In RS the above quantities are the same for all values of the replica 
index. Thereafter we will drop for notational simplicity the index a. The observables 
(JTS|l will be generated from the densities P^(<t,t). In general, from P^(ct,t) one can 
evaluate all higher order observables 

L ax-a m nv-7i = P $ ((T, t) a ai ■ ■ ■ O a ™T^ ■ ■ ■ T~* 1 (18) 

(TT 

Working out the simplest of these using the RS ansatz (fT4*)l gives: 

(TT 

= [ dhdrdt W,(h, r, t) ^^^^^), (19) 
xs J §v ' ; 1 + tanh(^)tanh(/5r)tanh(/3/i) /4 v ; 



{ / dhdrdtW^(h,r,t) 



tanh(/3/i) + tanh(/5r) tanh(/ft) 



-i 2 



1 + tanh(/3t) tanh(/fr) tanh(/3h)_ 
and similar expressions follow for the observables m%., q TT and q aT . 



>€ (20) 
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4. The self-consistent equation and the free energy 

4-1. Derivation of the self- consistent equation 

In order to arrive at a self-consistent equation for the densities W$(h, r, t) we first 
substitute the RS ansatz (JT4*j) into (fT2j) . Then, following 6J, we isolate the occurences 
of quantities of the form J2 a a a-> J2 a T a an d J2 a a aT a by inserting 



E 

m CT =— 00 
00 

E 



2tt 



2tt 



2tt 



drh T 



irha- (m CT -J] ct q ) 



2tt 



m T =— 00 
00 „ 27r 



1= V / — e ^ (9_E « CTaTa) 
^ Jo 2tt 

After some algebra we can take the limit n — >• in our equations and arrive at: 

dWz(h,r,t) e P hm °+Prm T +f3t q 

exp c( J dW£(ti,r',t') ^-^i^^/^O+^^^y/^^O+^a^y,*^^') _ ^ 



(21) 
(22) 
(23) 



(24) 



where we have used the notation dW^(h, r, t) = dhdrdt W${h, r, t) and we introduced 



K 1 (ti,r',t';Z-g) = -\og 



K 3 (ti,r',t';Z-£') = -\og 



(25) 
(26) 
(27) 



with 

Q aT = CTT (/i', rV; £ • £') = 4 cosh[/3/i' + a^ (£•£')] cosher' + r^(£ • £')] cosh[/3t'] + 

4sinh[/3/i' + (7^(£-0] siiMSr' + tM£ • £')] sinhltff] (28) 
c c 

Performing an inverse Fourier transform to (|24jl. expanding the right-hand side and 

integrating over the magnetisations m cr ,m T and the overlap q we obtain 



W € (/i J r,t) = X; 



00 n 1. 

e c 



k=0 



WdW^KruU)} 5 



1=1 



1 



1=1 



x 6 



k ~| r k "1 \ \ 



(29) 



Expression (J29|) is the final result from which to evaluate the densities W^(h,r,t). 
Although appearing a daunting numerical task it can be evaluated using the simple 
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algorithm of 'population dynamics' as described e.g. in [TSj. This relies on the local 'tree- 
like' structure of finitely-connected networks of spins and it consists of having a large 
population of triplets (hi, ri, ti) which are to be updated for a large number of iteration 
steps in the following way: One first selects a number k from a Poisson distribution of 
mean c. Then, one chooses I = 1, . . . , k triplets (hi, rj, t{) and I sublattices £ z at random 
and calculates the expressions appearing in the delta functions. Finally, one selects 
a new triplet (h, r, t) and a new sublattice £ at random and updates them using the 
calculated expressions. 

To get an idea about the profile of the densities W$(h, r, t) we have applied 
the previous algorithm to study the marginal densities W^(h) = f drdtW^(h,r,t), 
W£(r) = J dhdtW^(h,r,t) and Wj(t) = / dhdrW^(h,r,t). Firstly, we observe that 
close to the zero-temperature limit the marginals W^(h) and W^(r) become a collection 
of delta peaks; in this limit due to the absence of thermal noise the effective fields {h, r} 
become identical to the true local fields. Since the average number of connections per 
neuron is here a finite number and the couplings are discrete, the local fields are a 
multiple integer of 1/c || (see figure Q for typical profiles of the marginal densities). 
As one moves away from the T = regime the profile of the marginal densities WHh) 



0.15 




0.2 



0.15 



0.1 



0.05 - 



t I, I 





h 



Figure 1. Typical profiles of the marg inal densities W^{h) for p = 6, c = 4, T = 0.1 
(left picture, corresponding to the spin-glass region of figure |2J and for p — 2, c = 
5, T — 0.1 (right picture, corresponding to the retrieval region of figureEJ- Notice that 
since p is even the peaks are located at h = 21 /c with I integer. 



and W£(r) takes a non-trivial form until the paramagnetic phase is reached where one 
observes that W£(x) = WZ(x) = W£(x) = 5(x), as it should. We have numerically 
evaluated the above marginals for several regions of the parameter space and it is 
interesting to note that W]?(h) and W|(r) are identical (within the accuracy of numerical 

|| In fact, for even number of patterns p we have that the local fields present peaks at 21 /c with I 
integer, while for odd p we have l/c. 
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precision) \ whereas W^(t) = 8(t). The latter implies that our RS ansatz can be 
simplified further. However this result relies on numerical observations and a rigorous 
analytical proof appears somewhat hard (one cannot rely on an induction proof since at 
the first iteration step of ()29j) the fields t must take a non-zero value). Nevertheless, it is 
interesting to note that setting t = to completely decouples the density W^(h, r, t) 
to two identical densities 



e c cr 



k=0 



fc! 



Y[dxi Qt(xi)] 



1=1 



x 5 



x 



(30) 



~~ \ E ath ( tanh lP x i\ tanh[-(£ • £')] 

1=1 V ' C ' J / € i / e 

with x = {/i, r}. Note that this equation is now the same as the one that follows from 
the analysis of jH], as it should. 



4-2. The free energy 

Let us now express the free energy (JT3j) as function of the densities W^(h,r,t). The 
first term comprising (J13|) (energetic part) can be calculated without difficulty One 
uses the RS ansatz (JT3j) to replace the distributions P^(cr, r) by the densities W^(h, r, t) 
and subsequently one performs the spins summations and takes the limit n — > 0. The 
second term is slightly more complicated. First we expand the exponential function. 
This replicates the traces over the spins and sublattice variables. Inserting then for 
each of the replicated densities the RS ansatz (fTij) leads to 



log E exp 

(TT 



e c 



log^ 

fc=0 
n fc 



fc! 



(31) 



x 



II E II E ec'^'^° f « + ^ T ) + *^ + ^^ l5 » f i 



Performing the spin summations in the above equation and using the simple expression 
F(a a ,r a ) = Yl t jT^°>°^T,'r a F(. a i T ) t° relocate the occurences of a a and r a , we can take 
the limit n — > 0. The final result for the free energy (|T3~j) is then 



1 ^ e c c fc 



^ /,! 

r fc=0 



'J[dW e (h l} n,ti)] log 



i=l 



En 



=± i=i 



flg T (hi,ri,ti;$, ■ ^ 
Af(h h ri,ti) 



+1 k\J dW t(h,r,t)dW e (ti,r',t') 

% This is of course expected since the hamiltonian is invariant under the interchange of the two 
spin species. 
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J2tt>=± cosh[/3/i + ptr + ■ £')t'} cosh[/3/i' + pt'r' + £(£ • ^) T ]e^ r +^' r ' 



E TT '=± cosh[/3/i + /?tr] cosh[/3/i' + (3t'r]e. 



(32) 



5. Phase Diagrams 

5.1. Bifurcation Analysis 

The numerical observations concerning the densities W^(h,r,t) in section 0] support 
the equivalence between the parallel and sequential versions of the finite-c Hopfield 
model. In this section we will show further that one can analytically prove that 
the location of the second-order transitions between paramagnetic/spin-glass and 
paramagnetic/retrieval phases is identical to those of sequential dynamics. 

Our bifurcation analysis is similar in spirit to [6 a . First we note that W^(h,r,t) = 
S(h)S(r)5(t) solves (J29*j) for all £. This solution, as can be easily confirmed, corresponds 
to the high-temperature paramagnetic state where no recall is possible and m| = m£ = 
0. As the temperature is lowered from T = oo we expect non-trivial solutions to 
appear. To determine the transition temperature at which these occur we first assume 
that close to the transition J dW^(h,r,t)h e = 0(e e ), J dW^(h,r,t)r e = 0(e e ) and 
J dW^(h,r,t)t l = 0(e e ). We can then expand equation ({21} and identify term by term 
each expression in the resulting power series in the left- and right-hand sides of ()24jl . 
To second order, this one-to-one correspondence firstly indicates that integrals carrying 
the field t must vanish. We then obtain: 

J dW^h,r,t) h = c(J dW^(h,r,t) h tanh[^(£ • £')])<;' (33) 
J dWz{h,r,t) r = c(J dW?{h,r,t) r tanh[^(£ • £')])$' (34) 
J dW^h, r, t) h 2 - j dWz(h, r,t)h = c(J dW^(h, r, t) h 2 tanh[^(£ ■ £')]>*' (35) 



J dW(.(h,r,t)r 2 - fdWz(h,r,t)r = c(J dW € >{h, r, t) r 2 tanh[^(£ • £')]>£' (36) 

These four equations mark the different types of transitions away from the paramagnetic 
state. The first pair of equations will give us the transition from m| = and m£ = 
respectively to a non-trivial recall (ferromagnetic) state whereas the second pair will 
give us the transition to the spin-glass state. The resulting similarity in the first and 
second pair of the above equations is a consequence of the equivalence of the two species 
of spins in our system. 

To determine now the critical parameter values for which these transitions occur 
we need to evaluate the highest temperature for which the following TP x 2 P matrices 
have an eigenvalue equal to 1: 

il% = cp? tanh[^ • £')] Q# = cp? tanh 2 ^(£ ■ £')] (37) 
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These equations are identical to those found in [Hj. An elegant construction of the 
eigenvalues of matrices of the above form (which relies on exploiting the properties 
M^g = M(£ ■ £') and Q^> = Q(£ ■ £')) has already been given in [T5] . Here we state the 
final result determining the transition lines, identical to that found in [Oj: 



In general, the bifurcation analysis can not be applied to determine the spin- 
glass/retrieval transition line which must be computed directly from equations 
and ()19j) . This numerical task becomes increasingly difficult for large values of the 
connectivity parameter c (since one is required to evaluate 2 P sublattice densities where 
p ~ c close to the spin-glass/retrieval transition). For sufficiently small values of the 
connectivity parameter c (up to c = 7) we find that this transition line is identical 
to that of sequential dynamics. It is interesting to note that within replica-symmetric 
considerations, the c — > oo (extremely-diluted) phase diagrams of sequential and parallel 
Hopfield models are not identical, indicating that a critical c* exists above which W$(t) 
is no longer given by S(t). Determining this transition is a challenging numerical task. 

5.2. Results and comparison with simulations 

By numerically solving equations f)19|29j) we find that only one magnetisation component 
(also called pattern overlap) (|TB|) provides non-trivial solutions (i.e. only one pattern 
is retrieved while the others are zero.). For notational simplicity, we will from now on 
denote these non-zero solutions simply as m a and m T . 

Plotting solutions of (jHHjl and (fHTJj) in the (a,T) plane (with a = p/c) results in the 
phase diagram of figure 121 It has been plotted for c G {2, 3, 5, 7, 10} and it consists of 
three phases: a high-temperature paramagnetic phase with all observables identically 
zero, a retrieval phase with m a) m T ^ and q\\> = (with A = cr, r) and a spin-glass 
phase with m a = m T = and q\y ^ + . For T < 1 the spin-glass/retrieval transition 
is obtained numerically by direct evaluation of our observables. 

We see that already for c = 10 (upper line) the phase diagram resembles closely 
the one derived in the extremely-diluted model pQ where the transition lines are given 
by T = 1 for a < 1 (P— >R) and T = \fa for a > 1 (P— >SG). From the perspective of 
neural network operation, this shows that sparsely connected models retain the same 
qualitative features for a wide variety of synaptic connections per neuron and can retrieve 
information even for surprisingly small values of connections. 

In figureElwe present solutions of the sublattice overlap m a (fTH|) and m T showing the 
second-order transition from retrieval to paramagnetic states for the simplest non-trivial 

+ note that due to the inequality q ai a 2 ^ Qa 1 a 2 a 3 a i > • • • > we only need to examine q ai a 2 to identify 
the spin-glass phase. 





(39) 
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Figure 2. Phase diagram of the finite-connectivity Hopfield model with 
synchronous dynamics in the (ct,T) plane. Left: The paramagnetic/retrieval and 
paramagnetic/spin-glass transition lines as obtained from the bifurcation analysis, 
(from lower to upper lines c = {2, 3, 5, 7, 10}). Right: The c = 7 phase diagram with 
the spin-glass/retrieval transition line (obtained by direct evaluation of the pattern 
overlaps (jTT?|) using population dynamics). Markers correspond to integer p values 
(lines are simply put as guides to the eye). On the retrieval/spin-glass transition line 
m CT = m T = 0. 



case of p = 2. These have been drawn for c = {3, 5, 7} showing the effect of different 
degrees of connectivity on the retrieval success. To verify our results we have performed 
simulation experiments on the dynamical process (0) for a system size of N = 10, 000 
neurons (markers of figure EJ). These appear to be in very good aggrement with our 
theory. The simulation experiments also show that while Hebbian-type couplings (JH) 
lead to fixed-point stationary solutions, anti-Hebbian ones where Jy = — ^-(^ • £,•) lead 
to 2-cycles with merit) = —m (r (t+ 1) (and similarly for m T ). Also, evaluation of the free 
energy (J32j) as a function of temperature T shows that it is a monotonically decreasing 
function indicating that the entropy remains positive even for low temperatures. 

We have also compared our pattern overlaps with those derived from the analysis of 
[H] (which has as a starting the Ising 'sequential' Hamiltonian) and within the accuracy 
of numerical precision we have found that observables of the two systems are identical. 
This result, although somewhat expected given both knowledge from earlier neural 
network studies and, of course, the identity of our bifuraction results with those of 
[Hj, is also a bit surprising: in the process of solving our equations we introduced a 
3D RS 'effective-field' distribution (compared to the ID sequential one of [0]). As it 
turns out however the system in the process of updating fields by iteration of (f2"H|) 
effectively discards those which describe species-correlations and factorises W^(h,r,t) 
to W^(h, r) 8(t). Henceforth, our 3D RS field-distribution reduces to a 2D one. The 
equivalent treatment of a and r then ensures (as also in analytically simpler models) 
that equilibrium observables between sequential and synchronous systems are identical 
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in 




Figure 3. Pattern overlaps m a = rn T = m as functions of T = 1/(3 calculated from 
equations (|19|l and (|29[) . Parameter values are: p = 2 and c = {3, 5, 7} (from lower to 
upper lines). Markers correspond to simulation experiments of N = 10, 000 neurons. 



(this relation however ceases to exist for a certain large value c* since for c 
sequential and parallel phase diagrams are different |8,). 



oo the 




0.2 0.4 0.6 0.8 1 

a 

Figure 4. Pattern overlap = to^ = to m as a function of a (with c = 7) for 
T = 0.1 (circles) and T = 0.7 (triangles). Here we can see the first-order transition 
from retrieval to the spin-glass phase at a = 6/7 and a = 7/7 for T = 0.1 and T = 0.7 
respectively. 



In figure |U we have plotted the pattern overlaps and m% as a function of a for 
different values of the temperature (note that = m%) . We have set the average 
connectivity to c = 7 while varying the number of patterns p (therefore the number of 
pattern overlaps m' 1 also changes with a). From this graph we can complete the phase 
diagram (figure |2J), determining the location of the first-order transition from retrieval 
to spin-glass states. 
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In this paper we have studied equilibrium properties of attractor neural network 
models with finite connectivity in which neurons operate in a parallel way. This work 
is motivated by the interesting properties that the two types of dynamical models 
share in simpler (e.g. fully-connected) scenarios: There, and for a surprisingly large 
number of universality classes, one can prove analytically that the equilibrium states 
following from the two different dynamical rules are identical. Our starting point 
here is the Peretto Hamiltonian (j^J. We have followed on the footsteps of jB] to 
derive the transition lines in our phase diagram and expressions for the (sublattice) 
overlaps. The resulting phase diagram plotted in the (a, T) plane then consists of three 
phases: a high-temperature paramagnetic phase, a retrieval and a spin-glass phase. The 
transitions from the paramagnetic phase have been determined analytically while the 
first-order retrieval/spin- glass transition has been computed using population dynamics 
[T^] . Under our replica-symmetric considerations we have shown that the retrieval 
properties of the parallel finite-connectivity Hopfield model are identical to those of 
the sequential one. Comparison with numerical simulations for large system sizes shows 
excellent agreement. 

Many questions remain to be answered for neural networks of finite connectivity. 
Using the same framework (as initiated by |6 3 ) one can proceed further with the study of 
multi-state e.g. Q-Ising or Blume-Emery-Grifliths neural networks. A different appoach 
would be to study systems which consist of two species of operating units (as, effectively, 
here) but which explicitly interact with one another. This would lead us to an Ashkin- 
Teller-type neural network in which the phase diagram can be different. Non-trivial 
extensions would be to study the validity of the RS solution following the schemes in 
[iniEni- These will be the subject of a future work. 
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